Skip to content

Add a learner-local task mode - #259

Open
jserv wants to merge 4 commits into
mainfrom
tasktrial
Open

jserv wants to merge 4 commits into
mainfrom
tasktrial

Conversation

@jserv

@jserv jserv commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Task mode lets an instructor publish a PIN-encrypted problem set on a static HTTPS site, and a learner work one task with Jim on a locally run CodeTrial using their own Gemini and LiveKit credentials. The learner signs in with GitHub (device flow), enters the PIN, chooses one of the languages the task allows, accepts the announced rules, calibrates the camera and starts in fullscreen. Leaving fullscreen or the page, or looking away past the calibrated limit, ends the attempt as invalid; a camera, detector or connection failure ends it as interrupted; Finish or the deadline completes it with a learning review that carries no hiring verdict. Everything runs on a machine the learner controls, so the encryption and the rules deter rather than prevent, and every page and export says so. Interviews are unchanged, and the contract is docs/task-mode.md.

The series is four commits: stopping a dropped Gemini session's reader, which interviews benefit from as well; raising the mutation shard ceiling to 48, which a diff this size needs; task mode itself; and letting a task name the languages a learner may choose (Python when it names none; C, C++ and Java run on Compiler Explorer, as in interviews, and the page says so).

Automated checks

./scripts/test.sh covers task mode with no credentials:

./scripts/test.sh

The packaging tests need the cryptography package, which the gate takes from target/task-tools once a build has set it up.

On the rebased tip this passes with 1514 Rust tests, 1107 browser tests, the Python packaging and contract suites, clippy and formatting, and each commit builds on its own. The task mode suites are tests/unit/tasks/, tests/unit/livekit/tasks.rs, tests/web/tasks.rs, tests/browser/task-*.test.js, tests/test_task_package.py, tests/test_task_authoring.py and tests/test_task_mode.py; the Rust and Python sides validate the same fixtures under tests/fixtures/task-mode/ and decrypt each other's packages.

Trying it by hand

Building a set needs no credentials. scripts/task-package.py starts a bank from reference tasks, so nothing is written by hand: references lists the shipped examples (problem-bank/task-examples/, which show completion targets, required cases, custom checks and several languages) and then every catalog problem, and new copies one into a bank, creating it if needed, with the sidecar and set rules written out in full to edit in place. The tool finds a suitable Python itself, and the first build sets up target/task-tools with the package it needs:

scripts/task-package.py references
scripts/task-package.py new --bank course --from delimiter-closer
scripts/task-package.py new --bank course --from two-sum --id pair-sum \
    --languages python,javascript,cpp
scripts/task-package.py preview --bank course --task-id pair-sum
scripts/task-package.py build --bank course \
    --site https://USER.github.io/REPO --output publish

--id gives the copy its own id; it keeps the reference's origin, so the checks that keep a catalog problem's source out of what the learner reads still apply while you rewrite it, and new writes nothing unless the whole bank still validates. build asks for a six-digit PIN without echo, names the set after the bank's directory, takes the next version the output does not hold, and prints each task's assignment link. Publish publish/ as the root of a GitHub Pages site (any static HTTPS host works; the server refuses plain HTTP and redirects), so --site names where it is served.

Taking the task needs a Gemini API key, LiveKit credentials and a GitHub OAuth App with device flow enabled, whose public client ID goes in GITHUB_CLIENT_ID (without one, the page falls back to the browser sign-in, which also needs GITHUB_CLIENT_SECRET). Run cargo run -- web on the default loopback address and open a link build printed, such as http://localhost:3000/t/course/pair-sum?site=...&version=1. A first run goes through Setup and returns to the link. Then, in desktop Chromium or Firefox with a camera and microphone:

  • sign in, enter the PIN, and on pair-sum switch the language to see the starter, the editor label and the Compiler Explorer notice follow; delimiter-closer offers Python and JavaScript, each with its own completion target;
  • accept the rules, allow the devices, calibrate (a calibration without the typed line is refused) and Start;
  • run the public tests, ask for a hint, discuss a revision, and Finish to get the learning review and its export;
  • in further attempts, leave fullscreen, switch tabs, or look away past the limit (each ends as invalid), and stop the camera or reload the page (each ends as interrupted).

Task mode is off on a server reachable from elsewhere, one that records, or one that shares a fixed room; its routes then answer loopback_required.

Not done here: the recorded live pilot run the issue's gate asks for, against real LiveKit and Gemini with a published Pages set and real fullscreen and camera behaviour; docs/task-mode.md lists what it must cover. Release builds offer device sign-in only once a maintainer sets the repository variable CODETRIAL_GITHUB_CLIENT_ID to a GitHub OAuth App's public client ID; without it they fall back to the browser sign-in.

Close #242


Summary by cubic

Adds a learner-local task mode where an instructor publishes a PIN-encrypted problem set on a static HTTPS site and a learner works one task with Jim on a locally run CodeTrial using their own Gemini and LiveKit credentials. Interviews are unchanged, and the contract is docs/task-mode.md.

  • A task's sidecar lists the languages its learners may choose, defaulting to Python when it names none; the learner picks one before calibration, and the starter, targets, editor, runs, interviewer prompt, and result all follow it for the attempt. C, C++, and Java compile on Compiler Explorer, so the page says their runs leave the machine, and a server with it turned off offers only the task's other languages.
  • Breaking a rule (leaving fullscreen or the page, or looking away past the calibrated limit) ends the attempt as invalid; a camera, detector, or connection failure ends it as interrupted; Finish or the deadline completes it with a learning review that carries no hiring verdict.
  • One attempt runs per process, its result is held only for its own room and expires, and page actions route through the session so recovery sees that room's state. An assignment link opened before any config exists lands on Setup and reloads into the task once Setup saves.
  • Everything runs on a machine the learner controls, so the encryption and the rules deter rather than prevent, and every page and export says so. The instructor's tool starts a bank from a shipped example or a catalog problem, so a task is edited rather than written from nothing.
  • Release builds enable device sign-in only when a maintainer sets the repository variable CODETRIAL_GITHUB_CLIENT_ID; without it they fall back to browser sign-in. GitHub's signed-in user id is reported as the learner's identity only when GitHub issued it.
  • The first commit stops a dropped Gemini session's reader, which interviews benefit from too; CI installs the packaging tools in a venv outside the cached target directory, and the mutation shard ceiling rises to 48 so a diff of this size still meets the per-shard mutant budget.
  • Still missing from the issue's gate: a recorded live pilot run against real LiveKit and Gemini with a published set and real fullscreen and camera behaviour.

Written for commit 6d6f226. Summary will update on new commits.

View guided diff Turn on auto-fix

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

@jserv
jserv force-pushed the tasktrial branch 2 times, most recently from 5f5bd5f to c5b0b93 Compare October 9, 2026 15:41
cubic-dev-ai[bot]

This comment was marked as resolved.

@jserv
jserv force-pushed the tasktrial branch 3 times, most recently from a69ab16 to f2377b2 Compare October 10, 2026 00:42
jserv added 4 commits October 11, 2026 00:15
A JoinHandle going away does not cancel its task, so a live session
dropped without shutdown, on an early return or a replaced socket, left
its reader parked on the socket, holding the connection and its
provider slot open. Dropping the session now aborts the reader;
shutdown stays the orderly close.
Sixteen shards put a diff that adds a whole subsystem far past the
design load of 25 mutants a shard, and every shard ran out of its 35
minute budget before reporting anything. Forty-eight keeps the design
load for diffs up to twelve hundred mutants; shards beyond the
account's concurrent jobs wait in the queue rather than fail. The
planning comment still said the cap was four.
Every session today is a hiring interview: the client picks the problem
and duration, and the report judges toward hire or no hire. Task mode
lets an instructor publish a PIN-encrypted problem set on a static site
and a learner work one task with Jim on their own CodeTrial, ending in
a learning review with no hiring verdict. Everything runs on a machine
the learner controls, so the encryption and the announced rules
(fullscreen, page visibility and a calibrated look-away limit) deter
rather than prevent, and every page and export says so; an attempt that
breaks a rule or loses its page is recorded as invalid or interrupted
without asking a model. The instructor's tool starts a bank from a
shipped example or a catalog problem, so a task is edited rather than
written from nothing, and builds it for a site with each task's link.
Interviews are unchanged. The contract is docs/task-mode.md.

Close #242
Task mode admitted Python alone, though the catalog ships starters in
five languages and the page can run all of them. A task's sidecar now
lists the languages its learners may choose from, Python when it names
none. The learner picks one before calibration connects the room, and
the starter, the targets offered, the editor, the runs, the
interviewer's prompt and the result all follow it for the attempt. C,
C++ and Java compile on Compiler Explorer, as in interviews, so the
page says their runs leave the machine, and a server with it turned off
offers only the task's other languages. The task page now loads the
runtime configuration, without which it could not tell, and the
instructor's tool takes the list with a --languages option, dropping
targets marked in a language it leaves out.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a task mode for guided programming exercises with a learning review

1 participant